IRREDUCIBILITY AND SMOOTHNESS OF THE MODULI SPACE 
OF MATHEMATICAL 5-INSTANTONS OVER P 3 



I. COANDA 1 , A. TIKHOMIROV 2 , AND G. TRAUTMANN 3 



Abstract. We prove that the space of mathematical instantons with second Chern class 
5 over P3 is smooth and irreducible. Unified and simple proofs for the same statements 
in case of second Chern class < 4 are contained. 
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Introduction 



A mathematical instanton bundle is an algebraic vector bundle £ over P3(&), k an alge- 
braically closed field of characteristic 0, if it has rank 2, Chern classes C\ = 0, c<i = n > 0, 
and if it satisfies the vanishing conditions H°£ = and H l £{—2) = 0. The name was 
chosen in twistor theory in the 1970's, when holomorphic bundles on ^(C) with the 
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same properties were considered as counterparts of (anti-) self-dual Yang-Mills fields on 



the 4-sphere, see JT|, [27], [12] for reference. Let a mathematical instanton bundle with 
C2 = n be called n-instanton or (n, 2)-instanton for short. The isomorphism classes of 
n-instantons are the closed points of a coarse moduli scheme MI(n). Since its first con- 
sideration, it is an open problem whether MI(n) is smooth and irreducible for any n. An 
affirmative answer for n < 4 has been given in several papers, for each n separately, and 
recently Katsylo and Ottaviani proved smoothness for n = 5, see historical remarks in 
D 



The main result of this paper is that also MI(5) is irreducible, with a new proof of 
smoothness included. The method used also enables a simple and unified proof for all the 
previous cases n < 4, see Section ^. 

It is well-known that any n-instanton is the cohomology of a short complex H n ®0{— 1) — > 
N Cg> O — > H* ® 0(1), where H n and N are k- vector spaces of dimensions n and 2n + 2, 
respectively, also called the Horrocks construction. We consider also higher rank instan- 
ton bundles, called (n, r)-instantons, which can be constructed from the same type of 
complexes with the same H n but with dimiV = 2n + r, 2 < r < 2n. We prove that, given 
an (n, r)-instanton £, one can choose a linear form £ e H* such that with H n _\ = Ker(£) 
the induced complex H n _ x <g> 0(-l) -> N ® O -> H*^ ® 0(1) defines an (n - 1, r + 2)- 
instanton together with a complex 0(— 1) — ► ^ — » 0(1), whose cohomology is the 
original £ , see Section [3|. Together with technical details, this observation enables us to 
perform induction steps (n— 1, r + 2) -w (n, r) for n < 5 for irreducibility and smoothness. 
These induction steps are short for n < 4, see Section [5], while the case (4,4) -w (5, 2) is 
more elaborate. In Section R[ the induction step is explained in more details. The proof 
of smoothness in each case is achieved by proving that H 2 S 2 S^ = implies H 2 S 2 S = 
for suitable £. 



Notations 

• Throughout the paper k will be an algebraically closed field of characteristic zero 

• If E is a finite dimensional k- vector space, FE will denote the projective space of 
1-dimensional and G m E the Grassmannian of m-dimensional subspaces. 

• The invertible sheaf of degree d over FE is denoted CVe(g?) such that E* = 

H°(FE,OM±))- 

m P = FV = P3 will denote the projective 3-space over k for a fixed vector space V 
of dimension 4. We will omit the index F at the structure sheaf O = Op and at the 
invertible sheaves 0(d) = Op(d). 

• For an Op-module T we use the abbreviations T{d) = JF® P (d), R l T = H l (F, T) 
and ft\F(d) = dim fc H^d), and ext^J 17 , ^) = dim k Ext^F, Q). 

• If E is a finite dimensional fc-vector space, the sheaf of sections of the trivial vector 
bundle over a scheme X will be denoted E ® Ox = E ® k Ox and E (g) .F is written 
for (£ ® Ox) ®o x J 7 . 
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• If T respectively Q are coherent sheaves on schemes X resp. Y ', T Kl Q denotes the 
sheaf p*T ® q*Q onlxi.7 where p respectively q are the projections. 

• If T is a coherent sheaf on the scheme X and Y a closed subscheme of X we simply 
write Ty for the restriction on T ®ox Oy = T\Y . 

• Throughout the paper a vector bundle is a locally free sheaf of finite rank. If neces- 
sary, we will refer specifically to its bundle space as a fibration of vector spaces. 

• The Chern classes of a coherent sheaf T over P = P 3 are considered as integers 
and we also call the triple (ci(jF), C2{J r ), cz(T)) the Chern class of T . 



1. Instanton bundles 

1.1. Let M(2; 0, n, 0) denote the moduli space of semistable coherent sheaves on P3 of rank 
2 with Chern class (01,02,03) = (0,n, 0), which had been constructed by M. Maruyama, 
17| , [18] . It contains the open set M b (2; 0, n) of stable rank 2 vector bundles on P 3 with 



Chern class (ci, C2) = (0, n). Recall that a rank 2 vector bundle £ on P3 with Chern class 
Ci = is stable if and only if H°£ = 0. Then it is also simple, i.e. hom(£, £) = 1, see p0| . 
In this case the Riemann-Roch formula becomes 

ext l (£, £) - ext 2 (^, £) = 8n - 3, 

whereas Ext 1 (£,£) is isomorphic to the tangent space of M 6 (2;0,n) at [£}, the isomor- 
phism class of £. For large n the spaces M b (2;0,n) have many irreducible components 
and some of them have a much bigger dimension than 8n — 3. 



1.2. Definition: A symplectic mathematical instanton bundle with second Chern class 
n > and of rank r, or an (n, r)-instanton for short, is a locally free sheaf £ over 
P = WV = P3 with the following properties 

(i) £ has Chern class (0, n, 0) and 2 < r = rk(£) < 2n 

(ii) H°£ = and H 1 £(—2) = 

(iii) £ admits a symplectic isomorphism £ £*, (p* = —(p. 

Then £ must have even rank by (iii). In case rk(£) = 2 condition (iii) can be dropped 
because then the non-degenerate pairing £ <g> £ — > A 2 £ = O determines a symplectic 
form, which then is unique up to a scalar. 

If rk(£) = 2, then £ is stable by condition (ii). Therefore, the open part 

MI(n) C M fe (2;0,n) 

defined by the condition h 1 £(—2) = is the set of isomorphism classes of (n, 2)~instantons, 
also called n-instantons. 
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1.3. Conjecture: MI(n) is smooth and irreducible of dimension 8n — 3 for n > 1. 
This conjecture emerged in the late 1970's as n-instantons were considered as counterparts 



of self-dual Yang-Mills fields on the 4-sphere, see [|l|, [27], in twistor theory. 

The conjecture has been proved for n < 4. For n — 1 the space MJ(1) is the complement 
of the Grassmannian G(2, V) in F(A 2 V). For n = 2 the space MI (2) had been described 
by R. Hartshorne in [HJ as a smooth and irreducible fibration. The case n = 3 was proved 
by G. Ellingsrud-S.A. Str0mme in flQfl . W. Barth proved in [|3| that MI (4) is irreducible 
and J. LePotier in ETJ that M/(4) is smooth. Recently P.I. Katsylo and G. Ottaviani 
proved that also MJ(5) is smooth, see |15|. In this paper we prove that MI (5) is both 
smooth and irreducible. The method also yields simple and unified proofs for the previous 
results for n < 4. 



1.4. Further results on n— instantons: 

(1) Any n-instanton E is stable because H°E = and rk(£) = 2. Then the Grauert- 
Miilich theorem states that E has trivial splitting type, i.e. for a general line L in P3 the 
restricted bundle El is isomorphic to 20l, see 0. 

(2) A line L is called a jumping line of the n-instanton E if El — Ol(—cl) © Ol(o) with 
a 7^ 0, and this number is called the order of the jumping line. It is an easy consequence 



of the monad representation of E, see 2J3 that the set J(E) of all jumping lines of E is a 
hypersurface of degree n in the Grassmannian G of lines in P 3 . Moreover, n is the highest 
order possible for a jumping line. 

(3) For any n-instanton, h°E (1) < 2, see 0. The n-instantons with h°E(l) = 2 are called 
special 'tHooft bundles. They can be presented as extensions 



— 2CW-1) -» £ — O f-n, 1) -> 



where Q is a smooth quadric in P 3 , see ||13|| . It was shown there that MI(n) is smooth 
along the locus of special 'tHooft bundles, which has dimension 2n + 9. 

(4) A plane P in P 3 is called unstable for an n-instanton E if the restricted bundle Ep 
has sections, otherwise stable. If E is special 'tHooft, the unstable planes form a smooth 
quadric surface in P3, the dual of Q in (3). In [0] it was proved that an n-instanton E is 
already special 'tHooft if its variety of unstable planes has dimension > 2. 

(5) In |19| it was shown that MI(n) is smooth at any E with h°E (1) 7^ and that 
Ext 2 (£, £(—!)) = for such a bundle. The locus of these bundles has dimension 5n + 4. 



(6) P. Rao, [22J, and M. Skiti, |24 |, proved independently that MI(n) is even smooth 



along the locus of bundles E which admit jumping lines of highest order n. These bundles 



form a subvariety of dimension 6n + 2. Moreover, in [22] it is proved that MJ(5) is smooth 
at bundles which have jumping lines of order 4. 
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(7) L. Costa and G. Ottaviani, ||, proved that MI (5) is an affine scheme by describing 
the non- degeneracy condition for the monads, see |2[2|, as the non-vanishing of a hyperde- 
terminant. 



2. Instanton Bundles and Multilinear Algebra 



In this section we describe the monad construction of instantons from hypernets of 
quadrics. Throughout this paper 

H — H n 

denotes an n-dimensional /c-vector space, n > 1. We identify A 2 (H* <g> V*) with the space 
of anti-selfdual fc-linear maps 



H* (g V* 



Each uj G A 2 (H* ® V*) gives rise to a diagram 



H® V 



N* 



H*®V* 

u 

^ AT 



Q 



where N = is the image and Q = the cokernel of u>, and <p = (p w is the canonically 
induced symplectic isomorphism. Note that the rank of u is always even, uj is called 
non— degenerate if u(h <8> t>) ^ for any non-zero decomposable tensor /i ® t> in H ® V. 



2.1. Rank Stratification: The space A 2 (H* <g> V*) comes with the rank stratification 

A 2 (/T ® 7*) = tt An D fi 4 „-2 D...Dfi 2l 

where Vt 2m = n 2m (ff) = {w G A 2 (H* <g> V*) \ xk{u) < 2m}. This had been considered 
already in |25| by A. Tyurin. 

It is easy to prove by standard arguments that each fl2m is irreducible and smooth outside 
^2m-2 of codimension ( n ~ 2m j _ The tangent space at uj e Q% m \ ^2m-2 is the kernel in 

-> TM 2m -> A 2 (iT ® V*) -> A 2 Q -> 

which gives the dimension formula. There is the canonical decomposition 

A 2 (H* <g> 1/*) = (S 2 ^* <g> AV*) © (A 2 iT ® SV*), 

and on the first summand we have the (induced) rank stratification 

S 2 H* ® AV* = M 4n D M 4 „_ 2 D • • • D M 2 
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with M 2m = M 2m (H) = n 2m n (S 2 H* <g> A 2 V*). We let 

A = A(H) = {coe S 2 H* <g> A 2 V* | u is degenerate} 
be the subset of degenerate tensors. 

Remark: One can prove that A is a closed and irreducible subvariety of S 2 H* <g> A 2 V* 
of codimension 2(n — 1) for n > 2 and of codimension 1 for n — 1. 

2.2. The bundles £ w . To each oj G S' 2 !/'* <8> A 2 V* one can associate the complex 

H <g> Cp(-1) N®Ov^H*® C P (1), 

where a is the composition of N®O f ^H*®V*®0 ¥ and H*®V*®0 ¥ H*®O w (l). 
Then the following are equivalent: 

(i) a is surjective 

(ii) ip o a* is a subbundle 
(hi) cu is non-degenerate. 

Clearly (i) and (ii) are equivalent by definition. Now a is surjective if and only if for 
any (v) G PV the induced homomorphism iV — > iJ* ® (t>)* on the fibre is surjective or 
equivalently, that H <g> (v) —> H* ® V* is injective, which is the non-degeneracy. 

By this observation each oj G M 2m \ (^2m-2 U A), n < m < 2n, gives rise to an associated 
locally free sheaf 

£ w = Ker(a)/Im(9? o a*). 

This bundle is in fact an instanton bundle of rank 2m — 2n and with Chern class 
(ci,c 2 ,c 3 ) = (0,n, 0). This follows directly from the defining complex, also called the 
monad of S^. Rank and Chern classes can be computed from those of the sheaves of the 
complex as well as H°£ u = and iJ 1 £ aJ (— 2) = 0. Because £* is the cohomology of the 
dual complex, it follows that the symplectic isomorphism iV* N induces a symplectic 
isomorphism S* — ► E w . 

Remark: M 2n C A because otherwise a non-degenerate u G M 2n \ M 2n _ 2 would define 
the bundle = with non-trivial second Chern class. Therefore, In + 2 is the lowest 
possible rank for a tensor uo G S 2 H* ® A 2 V*, which is non-degenerate, and which then 
defines an instanton of rank 2. 

In the following lemma and proposition it is proved that any (n, r)-instanton arises by 
the above construction and that the isomorphism classes of (n, r)-instantons are in 1 : 1 
correspondence with the equivalence classes of the operators u G M 2n+r \ (M 2ri+r _ 2 U A). 
We use the following notation for 2 < r < 2n. 

M(n,r) := M 2n+r \ (M 2n+r _ 2 U A) 
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or 

M(n,r) := {uj G S 2 H* ® A 2 V* \ rk(oS) = 2n + r, uj non-degenerate}. 



2.2.1. Lemma: For any uj G M(n,r), there are isomorphisms 

H = H 2 (£ UJ ®Q 3 (1)) N = H\£ iU ®Q 1 ) Q^H l £^ 
H*^H x E w {-\) N* = H 2 ^®^ 2 ) 

which are compatible with Serre duality and the symplectic isomorphism £* = £ w , making 
the following diagram commute 

H ® V *- N* V - *- N *- H* ® V *- Q >- 

h 2 {e^® n 3 {i))®v ^ H 2 (£u,®n 2 ) < * H^s^^n 1 ) ^ ^(^(-l)) ® v* *- h 1 ^ o 

Proof. The homomorphisms of the bottom row of the diagram are induced by the natural 
homomorphism f2 3 (l) <g> V — >■ Q 2 and the exact sequences — > Q 2 — > A 2 V* (g> 0(— 2) — >■ 
fi 1 -»• andO -> fi 1 -> V*®0(-1) -> O -»• of the Koszul complex of V*®0(-1) -> C. 
Then the isomorphisms of the lemma and the commutativity of the diagram follow by 
tracing all data from the defining complex of £ u , using the functoriality of Serre-duality. 
See also || for monads of vector bundles on projective spaces. □ 



2.3. Proposition: (a) Let £ be an (n,r)-instanton with symplectic isomorphism £* — > 
£,2 < r < 2n. Then there exists a non-degenerate uj G S 2 H* ® A 2 V* of rank 2n + r such 
that (£J)^{S u ,j u ). 

(b) (SwjjJ) = (£u>',ju>') if an d only if uj and uj' are in the same GL(H)-orbit in S 2 H* ® 
A 2 V*. 

(c) The stabilizer Stab(uj) of uj in GL(H) is isomorphic to the automorphism group 
Avt(E u ,ju). 

Proof, (a) By |2.2.1| it is enough to show that the Beilinson spectral sequence of £ results 
in the complex 

-> H 2 {£ ® Q 3 {1)) ® C(-l) A H\£ ® Q 1 ) ® O A F x (^(-1)) <g> 0(1) -> 0, 
where a resp. f3 correspond to the natural homomorphisms 

H 2 (£ ® tt 3 (l)) <g> V -> # 2 (£ ® fi 2 ) i/ 1 ^ ® fi 1 ) 

resp. 
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which are Serre-dual to each other. Now for any coherent sheaf T on P„ there is the 
Beilinson I complex 

-> C- n (T) -> ► C Q (T) -> ► C n (JF) -> 

with terms C P {T) = ® H\F ® n j (J)) ® 0(-j), which is exact except at C°(.F) and 

p=i-j 

which has T as its cohomology at C°(T). In our case for T = S{— 1) this complex reduces 
in fact to 

-> # x (£ <g> ft 2 (l)) <g> C(-2) -> F 1 ^ <g> Q 1 ) <g> O(-l) -> (-1)) (8) C -> 

by verifying that the instanton conditions imply the vanishing of the other terms of the 
Beilinson complex. Moreover, the Koszul sequence — > Q 3 — > A 3 V* g) 0(— 3) — > fi 2 — > 
induces the isomorphism ® fi 2 (l)) = # 2 (£ ® ^ 3 (1)) because W£{-2) = for 

i = 1,2. Now £ is the cohomology of the complex 

-> ii 2 (£ ® fi 3 (l)) ® £>(-l) A tf 1 ^ ® ft 1 ) ® C A F x (f (-1)) ® 0(1) -> 0. 
It follows from the Riemann-Roch formula that 

x S(d)=r( d+ J) -n(d + 2) 



and from the instanton conditions that 

h 1 £(-l) = n and h}{8 ® ft 1 ) = 2n + r. 
Thus £ defines a non-degenerate u> G S 2 H* £g> A 2 V* via 

ii 2 (£ ® ft 3 (l)) <g> V -f ii 2 (£ ® ft 2 ) <- if 1 (5 <8> ft 1 ) -> H l S (-1) ® 

together with an isomorphism if* = 1), such that (£, j) = 



(b) and (c) follow from (a) and 2.2.1 by the isomorphisms of the complexes. □ 



2.4. Corollary; If £ u is simple, then StabcL(H)(u) = {±idn}- 



2.5. Remark: There is also a Beilinson II monad for an instanton bundle £ w . This can 
be treated in the same way as the above Beilinson I monad. Any to £ S 2 H* ®K 2 V* defines 
an operator H — > ii*®A 2 y* which is symmetric with respect to H and the exact sequence 
— > A" — > if* <g> V* — > Q — > as above. Now combined with the Koszul homomorphisms 

AV* <g> O(-l) -»■ ft x (l) and ft x (l) -»■ V* <g> 

we obtain the complex 

if ® 0(-l) Air®ft x (l) AQ®0^O. 
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If u is non-degenerate, this complex defines the (n, r)-instanton S^, where rk(u; 
This could be shown as in p. 2.1 



2n + r. 

73L but follows already from the following commutative 



diagram induced by to, which is a direct transformation between the two monad types. 





H <g> O(-l) — H* ® Q ® O 



N®0 — — H* ® V* g> O Q®0 



H* <g> O(l) — H* <g> O(l) 







2.6. The morphism M 2n+2 \ A -» MJ(n) 



According to Proposition |2.3| we are given a surjective map 1— > [S w ] from M2 n +2 s A to 



MI {n) whose fibres are the orbits under the action of GL{H). The map b is the underlying 
map of a morphism, because there is a universal monad over {M 2n +2 \A) xP with the 
universal family of n-instantons as cohomology. Then b is the modular morphism onto 
the open part MI(n) of the Maruyama scheme. It can be shown that b is a geometric 
quotient and a principal GL(if)/{±l}-bundle in the etale topology. However the latter 
fact will not be used in this paper. 



2.7. Criterion for irreducibility 

Since is irreducible of codimension ( 2n 2 -2 ) in A 2 (H* <g> V*), it follows that every 

irreducible component of M 2n +2 has codimension < ( 2n 2 -2 ) in S 2 H* ® A 2 V* or dimension 
> 3n(n + 1) — ( 2n 2 ~ 2 ) = n 2 + 8n — 3. From this observation we obtain 



2.7.1. Lemma: Any component of MI(n) has dimension > 8n — 3 and MI(n) is ir- 
reducible of dimension 8n — 3 if and only if M 2n +2 \ A is irreducible of the expected 
codimension ( 2n ~ 2 ) in S 2 H* <g> A 2 V*. 

For the proof use the following lemma. 
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2.7.2. Lemma: Let X — > Y be a morphism of reduced schemes, let Y be irreducible and 
assume that each fibre f~ l (f(x)), x G X, is irreducible of constant dimension e, and that 
each irreducible component of X is of dimension > dimY + e. Then X is irreducible. 

Proof. Let X x , . . . ,X p be the irreducible components of X. According to the theorem 
of the dimension of the fibres, f\Xi is dominant over Y for any i. Let X[ C Xi be the 
complement of the other components and let y G C\f(X-). Then 

r 1 (y) = u(f- 1 (y)nx i ) 

and the intersections f~ l (y) H Xi are distinct closed subsets of f~ l (y). As is 
irreducible, it follows that p — 1. □ 



2.7.3. Remark: In ||26|| , Example 1, A. Tyurin showed that M 2n and (consequently) 
M 2n+ 2 have not the expected codimension ( 2 ™) respectively f 2 "^ 2 ) in S 2 H* <g> A 2 V*. 



2.8. Criterion for smoothness 

Concerning the smoothness of MI(n), let £ be an n-instanton (of rank 2). Because 
[£] is a stable point of MI(n), this is a smooth point if Ext 2 (£,£) = 0. But since 
ext 1 (£,£) — ext 2 (£,£) = 8n — 3 the vanishing of Ext 2 (£,£) is also necessary if [£} is a 
smooth point and MI(n) has dimension 8n — 3 at [£]. On the other hand, we have 

Ext 2 (£ , £) = H 2 (£ y ® £) = H 2 {£ ® £) = H 2 {S 2 £). 

Now for any (n, r)-instanton we have the following 

2.8.1. Lemma: Leicj G S^if* ® A 2 V* 6e non-degenerate of rank 2m, n < m < 2n, and 
let £ — £ w . Then the following conditions are equivalent. 

(i) H 2 (S 2 £) = 

(ii) f2 2m and S 2 H* ® A 2 K* intersect transversally in u inside A 2 (H* <g> 1/*). 

(iii) M 2m is smooth at u of expected dimension 5n — 5n 2 + 8mn — 2m 2 — m. 

Proof. Transversality in (ii) means that the tangent space T aJ f2 2 m an d S 2 H* £g> A 2 V* 
span A 2 (if* (g> V*). Then (ii) and (iii) are equivalent by standard dimension counts, 
because f2 2m has codimension ( 4n ~ 2m ) anc [ j s smooth at cu. 

Next, let us recall a general fact. To any bounded complex K* of vector bundles on a 
variety one can associate its second symmetric power S 2 K* by decomposing K* ® K* 
according to the eigenspaces of the canonical involution on K* <g> K' . If K' has only three 
non-zero terms K^ 1 — > — ■> fC, then S 2 K° is isomorphic to 

-> A 2 ^ 1 -»■ K~ l <g> K -> S 2 K° © (fsT 1 ® fC) -> if ® K 1 -> A 2 ^ 1 -> 
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whose differentials are naturally deduced from those of K*. Moreover, if H l (K') = for 
i ^ and H°(K') = F, then H\S 2 K 9 ) = for i ^ and H°(S 2 K*) = S 2 F. Hence, in 
our case S 2 £ is the degree cohomology of the derived complex 

A 2 H ® 0(-2) ^H®N® 0(-l) -> (S 2 N ® H ® H*) ® O ^ N ® H* ® 0(1) -> A 2 iT <g> 0(2). 

The terms of this monad are cohomologically acyclic. Hence one can compute the coho- 
mology of S 2 £ just by passing to global sections and then taking cohomology. Particularly, 
one gets an exact sequence 

N ® H* <g> V* -> A 2 iT g> SV* -> H 2 (S 2 £) -> 0. 

Now condition (ii) is equivalent to 

T w 2m + (S 2 /T ® A 2 V) = A 2 (iT <g> V*), 

noting that u is a smooth point of fi 2m . Taking into account the exact sequences 

N(g)(H*(g) V*) -»■ A 2 (F* ® V*) -> A 2 Q -> 

and 

-> S 2 iT ® AV* -> A 2 (#* <g> 1/*) -> A 2 iJ* ® SV* -> 0, 

one finds that (ii) is equivalent to the surjectivity of the composed map 

N (g) H* ®V* ^ A 2 (H* g> K*) -> A 2 £P ® SV*, 

which is the map whose cokernel is H 2 (S 2 £). This proves the equivalence of (i) and 
(ii). □ 



Now, by the criteria for irreducibility and smoothness, it is clear that the conjecture [L3 
on MI{n) is equivalent to 



2.9. Transcribed Conjecture: fi 2n+2 arid S 2 H* <g> A 2 V* intersect transversally inside 
A 2 (if* eg) V*) along M 2n+2 \ A and this intersection is irreducible. 

Note, that in this conjecture, with A also M 2n has been extracted from M 2n+2 because 
M 2n C A. According to Tyurin's example the whole of M 2n+2 has components of excessive 
dimension for large n. 



2.10. Remark: Katsylo-Ottaviani gave in [IS] the following interpretation of the 
transversality condition. Dualizing the sequence with H 2 (S 2 £) as cokernel in the proof 



of Lemma |2.8.1| , we have the exact sequence 

-> H 2 (S 2 £)* -> A 2 H ® SV ^ N*(g)H®V. 
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Interpreting the elements of A 2 H ® S 2 V as anti-selfdual linear maps H* <S> V* — > H ® V 
and the elements of N* Cg> H ® V as linear maps N ^ H ®V , the morphism in the above 
sequence can be described by o i— ► (joti, where it is the inclusion of Im(cj) = N G H*®V* . 
Therefore, 

H 2 {S 2 S)* = {<t6 A 2 // g> S 2 V | a o w = 0} 



and the three conditions of |2.8.1| are equivalent to 

(iv) if a o u = for a G A 2 ii ® SV, then a = 0. 
In order to illustrate this point of view and for later use, we prove 



2.11. Proposition: M(n,2n — 2) = M 4n _ 2 \ (AU M 4n _ 4 ) is smooth and (obviously) of 
codimension 1 in S 2 H* <g> A 2 V* for n > 2. 



Proo/. Let uj G M 4n _ 2 \(AU M 4n _ 4 ) and a G A 2 ii <g> S 2 V such that a o uj = 0. Since 
both uj and <r are anti-selfdual, it follows that uj o a = 0. Now rk(u) = An — 2 implies 
rk(cr) < 2. One can now easily show (see e.g. | 25| , Proposition 2.1.1) that in this case 
a is decomposable, a — rj (g) /, with n G A 2 if of rank < 2 and / G S^V of rank < 1. 
Consequently, if cr ^ 0, then Im(cr) C H®V contains decomposable vectors, contradicting 
u & A. □ 



3. The Method 

Let E be an (n, r)-instanton and if* = if* = H X E^{— 1) as above with co> G S* 2 ii* ® 
A 2 ^* \ A of rank In + r. To any £ G ii* with kernel ii we can associate the restriction 
uj = uj^ = res^uj) of u to S 2 H* (g> A 2 ^* such that we have the diagram 

H®V — ^ H*®V* ■ 



H* <g> V 



Then rk(a>) < rk(a>). 
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The choice of £ gives rise to a diagram 
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H®0(-1) 0(1) 



H <g> C(-l) — - N <g> C #* <g> (9(1) 





£>(-!) #*®(9(1) 











such that [3, a constitute a monad for a bundle £ with an induced symplectic isomorphism, 



and such that j3, a are induced by u>. Then £ has rank r + 2 and second Chern class 
n — 1. Any splitting homomorphism of if ® — 1) — > 0(— 1) then induces a unique 
homomorphism O(-l) — > £ and by duality a monad 



whose cohomology is again £. This observation will be used to perform an induction 
(n — 1, r + 2) -w (n, r) for irreducibility and smoothness of the spaces of (n, r)-instantons 
for n < 5. 

Immediate relations between the tensors u and u; and the corresponding sheaves £ and £ 
are stated in the next two lemmata. 



3.1. Lemma: Let to be the restriction of uo for £ G H*. The following conditions are 
equivalent: 

(i) rk(u) = rk{uj) 

(ii) Im{uj) n (f ® V*) = 

(hi) TTie multiplication map H 1 £(—l) <g> V* — > ii 1 ^ restricts to an injection 



-> O(-l) -> £ -> 0(1) 







(iv) ii £ = 
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Proof. We have the diagram 



H <g> V H* ® V* 



H®V — ^ H* <g> V* — - Q 



AT* 



where iV is the image of cj, Ker(j*) = £ £§) V*, and // is isomorphic to the multiplication 
map H 1 £ (— 1) (g) V* — > iJ 1 ^. It follows that j* on is injective if and only if the image of 
uj has the same dimension as N, which proves the equivalence of (i) and (ii). The other 
statements are immediately seen to be equivalent to the injectivity of j* o u. For (iv), 
note, that iV — > H* ® V* corresponds to the right part of the monad of £. 

□ 

If the conditions of the lemma are satisfied, then £ is an (n — 1, r + 2)-instanton, provided 
r < 2n. In this case we have 



3.2. Lemma: Let uj G S 2 H*®K 2 V* be non-degenerate and rk(uj) = rk(uj). IfH 2 (S 2 £) = 
and H 1 ^!) = 0, then H 2 (S 2 £) = 0. 

Proof. Because £ is the cohomology of the monad O(-l) — > £ — > 0(1), S 2 £ is the 
cohomology of the induced monad, see proof of Lemma |2.8.1 , 

0@S 2 £ 



S(-l) 



The computation of H (S £) from this monad gives the vanishing. 



□ 



3.3. Construction lemma 

For n > 3 and 2 < r < 2n — 2 and for any £ G fP we consider the open subset 

M(n, r) ? := {uj G M{n, r) \ lm{uj) n (f <g> V*) = 0}. 

This is exactly the open subset of M(n, r) which is mapped to M(n — 1, r + 2) under the 
restriction map 

res 5 : u i-> uj^ = uj \ S 2 H* ® A 2 V*. 

We also use the notation 

M°(n,r) = {uj G M(n,r) | H l £^{l) = 0}. 
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Note that this set might be empty if h°£ u (l) - h}S u {\) = 4r - 3n < 0, while h 2 £ w (l) = 
/i 3 ^(l) = 0. Next we let 

M(n, r)' ( = {uj e M(n, r) f | res ( (uj) G M°(n - 1, r + 2)} 

be the inverse image. For the fibres of res^ we have the following lemma, which enables 
the induction. Note, that in this lemma the fibre R(uj) may contain degenerate u>'s. 

3.3.1. Lemma: a) For any uj G M(n — 1, r + 2) there is an isomorphism 

R(Q) ■ = {uo e S 2 H* <g> A 2 V* | res^uj) = Q, rk{uo) = rk{uj)} ^ Hom 0v {C Q) P (1)) 
where denotes the cokernel of the left part of the monad of and 

dimffomo p (C o ,0p(l)) =n-l + h°£^l). 

b) uj G i2(u)) is non-degenerate if and only if the corresponding homomorphism induces 
an epimorphism £ Q — > 0p(l). in £/ia£ case cj defines an (n,r)-instanton bundle £^ which 
is also the cohomology of the self-dual monad Cp(— 1) — > £& — > 0p(l) defined by the 
epimorphism. 

Proof, a) For £ fixed we may choose a decomposition H = H\ @ H such that £ induces 
an isomorphism H\ = k. Let u; decompose into 

H (g)V ^ N* ^ N >^ H* (g)V* 

with associated monad 

H <g> Op(-l) ^ iV <g> C P ^ if* ® Op(l). 
With Co) = coker(/9) we have the exact sequence 

-> C -> if* <g> Cp(1) -> 0. 
Any Mi G Hom(iV, if * ® V*) gives rise to a skew-symmetric operator 



Ml O ip O u\ 


U\ O if o -u* 


UO Lf O u\ 





(H X @H)®V ^ (H* © if*) <g> V* 

which factors also through ip by its definition, so that rk(u;) = rk(u>) and uj is symplectic. 
However, the component U\ o (p o u* : H ®V — > ifjf <8> V* is not necessarily skew with 
respect to V. This is the case if and only if uj G R(uj) or if and only if the composition 

if <g> Cp(-1) A TV <g> Op if* <g> O p (1) 
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of the associated sheaf homomorphismus is zero, or, if and only if U\ factors through C Q . 
Let Hom(iV, if j* ® V*)' denote the subsapce of Hom(iV, H± <g> V*) defined by this condition. 
Then U\ i— ► uj defines an isomorphism 

Rom(N,Hl®V*y *iR(w) 

because U\ \— > uj is injective since ip and u are surjective and because any uj e R{p) arises 
in this way. On the other hand, the factorization of U\ : N ® Op -» — » if* (g> 0p(l) 
defines an isomorphism iti «-> 7 between Hom(iV, H± <S> V*)' and Hom(C , H* (g) (9p(l)). 

b) By a) any uj e R(uj) gives rise to a diagram 



H* <8> Cp(1) 




H <g> P (-1) iV ® C P C 





with 7r induced by u% or ax- It follows that a = a x + a : iV ® C P — > (H* © /f*) <g> Cp(l) is 
surjective if and only if 7r is surjective. This proves b), because the surjectivity of a (as 
right part of the monad of uj) is equivalent to the non-degeneracy of uj. In that case the 
induced sequence 

#1 ® Cp(-l) — E a A tfj* ® Op(l) 
is also a monad for the bundle □ 



3.3.2. Corollary: IfM°(n— 1, r+2) zs irreducible of the expected dimension 3(n — l)n — 
( 2n " 2 4 " r ) ; then also M(n , r)^ is irreducible of the expected dimension 



3n(n + 1" 



2n 



if it is not empty. 



Proof. The dimension of any component of M(n, r) is > then the expected dimension by 
|2.7| . Because the fibres of res^ have constant dimension by |3.3.1| , the corollary follows from 



Lemma 2.7.2 



□ 
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3.4. Lemma: If M(n, r)t 7^ 0, then for any other rj 7^ 0, 

M(n,r)' ( nM(n,r)' v ^ 0. 

Proof. For c<Jo G M(n, r)^ we consider the set 

U = {rj e H* I Im(wo) n (77 ® V*) = 0, /^(l) = 0} 

where S v denotes the bundle obtained from uj° by restriction to the kernel of rj. Because 
(£ v ) is a flat family on the open set of 77's defined by Im(u;o) fl (rj <E> V"*) = 0, the semicon- 
tinuity theorem implies that also Uq is open. Because £ G Uq, there is a £' G C/o which is 
independent of £. Then cl> G M(n, r)^ fl M(n, r)t,. 

Let now 77 7^ be arbitrary in if*. There is a transformation g G GL(H) such that 
g*£ = i and = 77. Then 

(g* ® id y ») o c;o o (p id y ) G M(n, r)£ n M(n, 7%. 

□ 



3.5. The induction: Suppose now that M°(n — l,r + 2) is irreducible and that also 
M(n, r)l 7^ for some £ 7^ 0. Then by Lemma |3.4| the union 

U M(n,r)' f 

is an irreducible open subset of M(n, r) of the expected dimension. If, in addition, M°(n — 
1, r + 2) is smooth (as transversal intersection), then by Lemma |3.2| also H 2 S 2 £ U) = for 
any u; in the above union. Concerning rank 2 instantons, we shall prove that 

M(n, 2) = U M(n, 2) € 

for n > 3. For n = 3, 4 we shall even prove that 

M(n, 2) = UAf (n, 2)£. 

This will give a unified proof of the instanton conjecture for n < 4. 

For n = 5, however, we are at present not able to prove that for any uj G M(5, 2) there 
exists a £ 7^ such that H X £&(X) = 0. But we shall prove the weaker result, that there 
exists a £ 7^ with h x £u{l) < 1. This is already sufficient to prove the conjecture for 
7i = 5 in the sequel. 

For n = 6 one might hope that UM(6, 2)^ is the complement of the subvariety of 'tHooft 
bundles (defined by h°S(l) 7^ 0). However, for n > 6 the present induction method 
doesn't seem to work anymore. 
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4. On Jumping Lines of ^-Instantons 



In this section we are going to prove the following Proposition O which enables us to 
choose £ G H* for any n-instanton £u,n > 3, such that := £ Q is again an (n — 1,4)- 
instanton, i.e. H°£^ = 0. For n > 5 the proposition allows us to choose a second rj G H*_ ± 
such that also H°{£^) n = 

In order to prepare the proof we include the following lemmata on jumping lines of arbi- 
trary instantons. 



4.1. Lemma: Let £ be a stable rank 2 vector bundle on P 2 with Chern class {01,02) = 
(0, n), n>2, let Li,L 2 be distinct lines and let ai,a 2 > be defined by £i v = Ol v (o>u) ® 
Ol v (-a v ). Then 

(a) ai + a 2 < n 

(b) If ai + a 2 = n, and a\ >2,a 2 > 2, then £ can be realized as an extension 

0->O Pa (-l) ^£^ J z (l) ^0 
with Z C Li U L 2 , a 0- dimensional subscheme of length n + 1. 

Proof, (a) There is the natural exact sequence 

-> Li ul 2 -> Olj © O l . 2 -> (9 Liri L 2 -> 0. (1) 
Tensoring this with £ and taking sections, this implies 

a 1 + a 2 < h°£ Ll uL 2 - 

On the other hand, the sequence 

-> £(-2) -> £ -> £ LlULa -> (2) 

implies the exact sequence 

0^H°£^ H°£ LlUL . 2 -> H x £{-2) -> iJ 1 ^. 

Because £ is stable, = and h 2 £{-2) = h°£(-l) = 0, so that h x £(-2) = nhy the 
Riemann-Roch formula, and then /i°^L 1 uL 2 — n - 

(b) According to the proof of (a), if ai + a 2 = n, then ai + a 2 = /i ^L 1 uL 2 — n an d 

H°£l! © H°£l 2 —> H°£ LinL2 
is surjective. It follows that then also 

tf°£ Ll (l) © #°£ i2 (l) - ^L in L 2 (l) 

is surjective and from this, that 

h°£ LlUL2 {l) = n + 2. 

Using sequence (2) with £(1), we obtain h°£(l) > 2. Then there are exact sequences 

0^Op 2 (-l)^£^J z (l)^0 
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with O-dimensional subschemes Z of P 2 of length n + 1. Tensoring (1) with Oz and taking 
lengths, one obtains 

length(Z n (Li U L 2 )) > length(Z n L x ) + length(Z n L 2 ) - 1 

(ai + 1) + (a 2 + 1) - 1 
= n + 1 

length(Z). 

Then Z C L\ U L 2 as schemes. □ 



4.2. Lemma: Le£ £ be a semistable rank 2 vector bundle on P 2 with Chern class (ci, c 2 ) = 
(0, n), n > 2. T/ien 

(a) For odd n > 3 or even n > 6, £ has only finitely many jumping lines of order > n/2. 

(b) For n = 2 or 4, £ has at most one jumping line of order > n/2. 

Proof. Note, that semistability in this case means that H°S(—1) = 0, see ||20|| . Let firstly 
£ be stable with H°£ = 0. Then [4.1| , (a), implies that £ has at most one jumping line of 
order > n/2. Suppose n > 4 is even. According to |4.1| , (b), if £ has two jumping lines 
Li, L 2 of order > n/2, then there is an extension 

0^O ¥2 (-l)^£^l z (l)^0 

with Z C L\ U L 2 of length n + 1. If L is any other line, then 

length (Z n L) < length (L n {L x UL 2 )) = 2, 

and it follows that £ L = L {a) © L (-a) with < a < 1. 

Now assume that £ is properly semistable. In this case we have an exact sequence 

-> C Pa -> £ -> J z -> 

with a O-dimensional scheme Z of length n. If L is a jumping line of £ order > n/2, 
then also length [Z fl L) > n/2 by the exact sequence tensored with 0£. If £ should 
have infinitely many jumping lines of order > n/2, then there is a point 16Z such that 
infinitely many such jumping lines meet Z exactly in x. Then length (Oz, x ® ®l,x) > n /2 
for infinitely many lines. It follows that the germ [Z, x) is defined by two equations 
f,g £ Op 2jX (Z is a locally complete intersection) with mult(/), mult(o) > n/2. Then 

n = length(Z) > length(0^ s ) > mult (/) mult (g). 

This is not possible for odd n > 3 or even n > 6. For n = 2 or 4 the same kind of argument 
as in the last part of the proof of |4.1| , (b), shows that £ has at most one jumping line of 
order > n/2. □ 
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4.3. Proposition: Let £ be an n-instanton on P 3 , n > 2. Then the set of jumping lines 
of £ of order > n/2 has dimension < 1. 



Proof. Let G denote the Grassmannian of lines in P3 and consider the incidence diagram 
of lines in planes. Let £ be the set of jumping lines of order > n/2. According to 4.2 



the projection g|p _1 (S) has finite fibres. Suppose dimS > 2. Then g(p _1 (E)) = P3. Let 
then Pq be a stable plane for £. It must contain a jumping line Lq of order > n/2. For 
a general plane P containing Lq, £p is stable and, according to P~T| , (a), L is the only 
jumping line of order > n/2 contained in P. On the other hand, for any P containing 
Lq the bundle £p is semistable. Hence, by [4.2| , P contains at most finitely many jumping 
lines of order > n/2. Consequently, there are only finitely many jumping lines L of order 
> n/2 meeting L . This means that T Lo (G) D S is a finite set, where T Lo (G) denotes the 
geometric tangent hyperplane to <G at L in P 5 , whose intersection with G is the cone of 
lines meeting L . This contradicts dimS > 2. □ 



4.4. Remark: One can show, using the method of R. Strano and M. Green, as in ||, 
that a non special 'tHooft n-instanton £ , n > 5, satisfies h°£p(l) < 1 for a general plane 
P. Then, using the above arguments, one deduces that the set of jumping lines of £ of 
order > n/2 is at most 1-dimensional. But this improves |4.2j only for even n > 6. 

From the above statements on jumping lines we can now derive the following proposition, 
which is the key of the induction process of this paper. 



4.5. Proposition: a) Let £ be any n-instanton. If n > 3, then for a general £ G 
H l £{— 1) the multiplication map £ <S> V* — ► H 1 £ is injective. 

b) If n > 5 ; then for a general 2-dimensional subspace U C H 1 £ (— 1) the multiplication 
map U <S> V* — » H 1 £ is injective. 

Note, that the condition in a) means that Im(u;) fl (£ <8> V*) = 0, see |3TT| , because the 
multiplication map is isomorphic to H <S> V* — > Q. 

Proof, a) The right part iV ®0 — > if* ® 0(1) of the monad of £^ corresponds to the exact 
sequence -> iV -> if* ® V* A Q -> 0, where /i is isomorphic to the multiplication map. 
If P C P is a plane with equation z G V*, we obtain the exact sequence 

-»■ H°£ P -> JV -> ii* ® V7<«> 

and, therefore, 

H°£p = Nf](H*®z). (1) 
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On the other hand, if h°£p ^ 0, we are given an exact sequence 

-> P -> £ P -> J z -> 

where Z is a 0-dimensional subscheme of P of length n, because h°£p(— 1) = 0. It follows 
that 

< /i°£ P < 1. (2) 

We let now P(iV)i denote the set of decomposable classes (£<8)z) G F(H* ® V^*) which are 
contained in P(N), so that 

p(iv)i = P(iV) n 5i 

where Si is the image of the Segre embedding FH* x PV^* C F(H* <g> V*). There are the 
two projections 

FH* A- F(N) 1 A FV*. 
The isomorphism (1) implies that for any z^Owe have 

F(H £p) = q^((z)), 

where P is the plane with equation z. Therefore, the image of qi is contained in the 
subvariety JP(£) C PV^* of unstable planes of £. 

Now (2) implies that P(iV)i is isomorphic to JP(£). Because dim JP{£) < 2, see 0, it 
follows that pi(F(N)i) has dimensions < 2. This proves a) in case n > 4. If n = 3 and 
£ is not special 'tHooft, then dim JP{£) < 1 by |L~4| , (4), so that a) is true in that case, 
too. But in case n = 3 and £ is special 'tHooft, the claim follows from remark |4.6| below. 

b) For the proof of part b) we consider the intersection 

P(JV) 2 = F(N) n S 2 

with the secant variety of Si in F(H* <g) V*), such that F(N) 2 consists of all elements of 
type (£i ® z\ + £ 2 ® z 2 ) which are contained in F(N). Then we have the two projections 

G{2, H*) ^ F{N) 2 \ P(JV)j ^> G(2, V*). 

We have to show that a general 2-dimensional subspace K C if* satisfies iVn(.K"(8> V*) = 0. 

Now JV n ® 1/*) ^ if and only if either F(K) n pi(P(iV)i) ^ or K e lm(p 2 ). 

Because dimpi(P(JV)) < 2 by part a), it follows that the set of K with F(K)r\p 1 (F(N) 1 ) ^ 
has dimension < 2 + (n — 2) = n. By assumption, dimG(2, H*) = 2(n — 2) > n + 1 
for n > 6 and = n + 1 for n — 5. Therefore, it remains to prove that dimlm(p 2 ) < 
n + 1 for n > 6 and dimlm(j> 2 ) < 5 for n = 5. 

In order to derive the estimates, we consider the fibres of q 2 . Let W C V* be any 
2-dimensional subspace with the dual line L C PV. Then 

iV n (P* (g> VK) = H°£ L 



22 COANDA, TIKHOMIROV, AND TRAUTMANN 

as can easily be derived from the monad description of £. On the other hand, 

q-\W) = ¥(N n H* <g> W) s P(JV)i 

by definition of P(iV) 2 . Therefore, 

dimg^ 1 (W) = /i 5 J ;-l. (3) 

Let G(2, V*) D Ji D J2 D • • • D J n be the filtration by the sets Jfc of jumping lines L of 
order — 1 > k. It follows that 

dimg 2 ~ 1 (Jfc \ Jk+i) < 3 + k < n 

for < n — 3. For k = n — 2 

dimg 2 " 1 (J„_ 2 \ J n -i) < 1 + (n — 2) = n — 1 



because of Lemma |4.3| since n — 2 > n/2, and finally 

dimg 2 " 1 (J n _ 1 \ J n ) < n , dimg 2 ~ 1 (J n ) < n + 1. 

Totally we have dim(P(A^) 2 \P(JV)iJ < n + 1, which is sufficient for n > 6. In case n — 5, 
dim(P(A^) 2 \ P(iV)i) = 6 is only possible if dim J 5 = 1. In that case £ is a special 'tHooft 
bundle, and it follows by the direct argument in remark fL6], that there are 2-dimensional 
subspaces K C H* with JV n {K <g> V*) = 0. This proves b). □ 



4.6. Remark: Let £ be an (n, 2)-instanton on P3 with n > 2m + 2 > 3. One can prove, 
along the same lines, that then, for a general m-dimensional subspace K C H 1 £(— 1) the 
multiplication map K ® V* — > iJ 1 ^ is injective. If £ is a special 'tHooft bundle, this can 
be verified by the following direct argument. Let z , . . . , z 3 be a basis of V*. Then bases 
of iV and H* can be chosen such that the operator iV — ► H* <g> V*, i.e. the right part of 
the monad of £, can be represented by the matrix 

f Zq Zi Z 2 Z3 ^ 
Zq Z\ Z2 Z3 

\ z z l z 2 z 3 / 

If n = 2m + 1 or 2m + 2 and ei, . . . , e n is the corresponding basis of H*, then 
if = span(e 2 , e 4 , . . . , e 2m ) satisfies N f] (K ® V*) = 0. 

For later use, we consider the space n(£) C G(2,H*) of all 2-dimensional subspaces 
K C H* which contain £. We have: 

4.7. Lemma: Lei n = 5 and uj £ M(5, 2) snc/i £/iat £ w zs not special 'tHooft. Then for 
a general £ £ _£P taere are closed subsets T x C T 2 C n(£), dimT^ < 1 and T 2 a surface, 
such that 
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(i) N n (if <g> V*) = for K G n(£) \ T 2 ; 

(ii) dim fl (if <8> V*) = 1 and N fl (if (g> V*) contains no non-zero decomposable vector 
of H* ®V* for if GTaxTi. 

Proof. We use the previous notation and consider the morphisms 

P(iV)i A P(iT) and P(A^) 2 \ P(JV)i ^ G(2, H*) =: G. 



a) Let Y 1 C G denote the subset of those if G G for which iV fl (if <g> V*) contains 
a non-zero decomposable vector or, equivalently, P(if) H pi(F(N)i) ^ 0. Because 
dimpi (P(A r )i) < 1, the set of lines P(if) in F(H*) with this condition is closed and 
of dimension < 4. 

b) Let Y 2 C G be the set of all if C G for which AT n (if ® V*) ^ 0. If /C C ii* <g> O g 
denotes the universal subbundle, we consider the homomorphism 

obtained as the composition of fC ® V* C H* ® V* ® Cg an d the multiplication map 
H* ® V* — » Q with kernel A/". Then y 2 is the determinantal subvariety of <fi. Because, for 
a general if we have A" fl (if <g> V*) = by Proposition [4.5| , I2 is a hypersurface in G, 
dim Y" 2 = 5. By definition of Y 2 we have Im(p 2 ) C Y 2 , and, furthermore, 

Y 2 = Y 1 Ulm(p 2 ). 



c) For if G Y 2 \ Y\ the fibre p 2 1 (K ) consists of vectors £1 <E> £1 + £ 2 ® z 2 G A/", where £1, £ 2 
is a basis of if and Zi, 22 £ V*, i.e. 

Pa^if) = P(ivn (if <g>\/*)). 



Now, by the proof of Proposition [4.5| , p 2 (P(AQ 2 \ P(AT)i) has dimension < 5. It follows 



that, for a general if in each component of Y 2 \ Yi, the fibre p 2 (K) is a point, or, 
equivalently, A" fl (if ® V*) is 1-dimensional. Then the subvariety Kj' C 1^, defined by 
dim N n (if <g> V*) > 2, has dimensional < 4. Let Z X = Y\U Y{. Then dim Zi < 4 and for 
if G Y 2 \ Zi the intersection A" fl (if (g) V*) is 1-dimensional and contains no non-zero 
decomposable vector. 

d) Because dimZi < 4, the general 3-space IT(£) meets Z\ at most in dimension 1. Let, 
now, T\ = n(£) fl Z\ and T 2 = II(£) fl Y 2 . We may assume that IT(£) G: F 2 by Proposition 
^4.5| , b), so that T 2 is a surface in IT(£). Then Ti C T 2 satisfy the properties of the 
lemma. □ 
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5. The varieties M(n,r) for n < 4 

We first note that M(n, 2n) is an open subset of the affine space S 2 H* ® A 2 V*. In fact, 
it is the complement of the hypersurface M 4n _2- In this case, if u> € M(n, 2n), the bundle 
£„, is the cokernel in 



O^H® Op(-1) -> if* ® ^(1) -»• £ w -»• 
see 2J3 with Q = 0. Then H l Eu{i) = for i > and we have 

M°(n,2n) = M(n,2n). 



5.1. Proposition: For n = 2, M 6 = M 4n _ 2 (if) an irreducible hypersurface in S 2 H*< 
A 2 V*. 



Proof. Note that M6 \ (M4 U A) had been shown in |2.11| to be a smooth hypersurface, 
and Me is a homogeneous hypersurface in S 2 H2 <S> A 2 V*. 

In order to prove that it is irreducible, it suffices to prove that it is non-singular in 
codimension 1. Now A = A(H 2 ) is closed and irreducible in S 2 !!^ ®K 2 V* of codimension 
2. We have M4 C A and hence M§ \ A is smooth. It is therefore sufficient to find an 
oj G A \ M4 which is a smooth point of Mq. For that choose an isomorphism H2 — k 2 
and let H2 — > H% ® A 2 V* be presented by the matrix 

x 2 A x 3 x 2 A £ 4 

x 2 A14 Xi A x 2 + x 3 A £ 4 

where basis of V*. One can easily check that this u is degenerate at the 

point X2 = X3 = x^ = and that rk(a>) = 6. Moreover, u is a smooth point of M 6 , using 



the argument of Katsylo-Ottaviani as in the proof of Proposition [2.11| . □ 



5.2. Corollary 1: (Hartshorne) MI(2) is smooth and irreducible of the expected dimen- 
sion 13. 

Proof. M(2, 2) = Mq \ (M4 U A) is a smooth transversal intersection and irreducible. By 
!9| the results follows. □ 



5.3. Corollary 2: (Ellingsrud - Str0mme) MJ(3) zs smooth and irreducible of the ex- 
pected dimension 21. 



Proof. As noted above M°(2,4) = M(2,4), and this is irreducible. By Proposition 4J3 



M(3, 2) is the union of the open sets M(3, 2)g which are equal to M(3, 2)^ and which 
are transversal intersections and irreducible. The result follows now as in the previous 
case. □ 
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In order to treat the case MI (4), we prove the following 
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5.4. Lemma: For n = 3, M 10 = M in _ 2 (H) is an irreducible hypersurface in S 2 H* 
A 2 V*. 



Proof. By gTTT] M 10 is a hypersurface in S 2 HI <g> A 2 V* and M(3, 4) = M w \(AU M 8 ) is 



smooth. Because for n = 3 we have Mq C A, M(3, 2) = Mg \ A and hence A U M§ 
A U M(3, 2). Now A is irreducible in S 2 H^ ® A 2 V* of codimension 4, see remark in |2Tl 



and M(3, 2) has codimension 6 in S 2 H^ ® A 2 V* by corollary |y| It follows that M 10 is 
smooth in codimension 1 and so M\q is irreducible. □ 



5.5. Corollary: M(3,4) is irreducible and a (smooth) transversal intersection of expected 
dimension 45. 



5.6. Proposition: (Barth, LePotier) MI (4) is smooth and irreducible of the expected 
dimension 29. 

Proof. 1) Recall that the open subsets M(4, 2)£ C M(4, 2) are defined by 

M(4,2)£ = {uG M(4,2) I u> G M(3,4) and = 0}, 

where = res^(u;). In this proof we write T = which depends on the choice of £. 
By |J M(4, 2)' ? is irreducible. It is also smooth: by R 2 S 2 T = for u; G M(3, 4) 
because M(3,4) is a smooth transversal intersection of the expected (co)dimension. It 
follows from Lemma p.2| that also H 2 S 2 £ U) = for u G M(4, 2)^, hence again by |2.8.1 
M(4, 2)^ is smooth at any of its points and of expected dimension. 

2) It suffices now to prove that 

M(4, 2) = u M(4, 2)' e . 

Let u G M(4, 2). By |6j there is £ G H 1 £ U) {-1) such that H 1 ^!) = for any plane P 
in P 3 , provided £ w is not special 'tHooft. In case n = 4 we obtain /i x JF = 2. Then the 
operator if x jF ® V* — > can be presented by a matrix 



UH ... f i m 
V21 • • • f2m 



with m = /i 1 ^ r (l) if ^ 0. Let now P be a plane with equation z = and such 

that z(un) = z(v 2 i) = 0. Then 
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cannot be surjective and it would follow that H 1 J r p(l) = cokeiA(z) 7^ 0, a contradiction. 
This proves that u G M(4, 2)£ for the chosen £. If, however, 8 W is special 'tHooft, the 
pairing H 1 8 U) {— 1) <8> V* — > H 1 8 UJ can be presented by a 4 x 6 matrix 

ViV 2 

\ V 3 V 4 J , 

where Vi, . . . , t> 4 is a basis of V, see ||. Then, choosing £ = (0, 1, 0, 0) G k 4 = 1), 
the resulting homomorphism i7 1 ^ r (— 1) ® U* — > H X T is presented by the matrix 









Vl 




v 3 


V 4 



It follows that H^il) = because of the exact sequence H 1 ?^-!) ® A 2 V* -> i? 1 ^ ® 
V* -> #^(1) 0. □ 



5.7. Remark: Concerning MI[b) and M(4,4), it follows from [O] that the open set 

U M(4,4) £ 

is irreducible and a (smooth) transversal intersection of expected dimension, because 
M°(3, 6) = M(3, 6) is smooth of expected dimension. This will suffice to prove that MI (5) 
is smooth and irreducible of dimension 37. Using the method of Katsylo-Ottaviani as in 
the proof of |2.11| , one can show that M(4, 4) is smooth of the expected dimension. It is 
an open question whether it is also irreducible. 



6. A TECHNICAL RESULT ABOUT 5-INSTANTONS 



6.1. Proposition: Let 8^ be a 5-instanton on P3. Then for a general £ G H 1 8 u) {— 1) the 
associated rank-4 bundle T ' — satisfies /i 1 J r (l) < 1. 



For the proof we need the following lemmata on the vanishing of H 1 J-'l(1) and H l Tp{l) 
for lines and planes. 

6.2. Lemma: Let 8^ be a n-instanton, n = 4 or 5, and assume that 8^ is not a special 
'tHooft bundle {h Q 8^{\) < 2). Then, for a general £ G H 1 £ tl) {—1), the bundle T ' = 8 a has 
the property that i/ 1 jF i (l) = for any line L in¥ 3 . 
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Proof. Let £ = £ UJ . For any plane P in P 3 we have H 1 S( — 1) ^ H 1 £p(—1). Then for 
any line L C P the restriction H 1 £(— 1) — > H 1 £l(— 1) is surjective because H 1 £p(— 1) — > 
H 1 £l(— 1) is surjective, since H 1 £p(—2) = 0. Because £l is the cohomology of the monad 

0^O L (-l)^F L ^O L (l)^0, 

we obtain 

H^ L {1) = H l £ L {l)/i L - H O L (2), 

where ^ denotes the restriction of £ in H 1 £ L (— 1). On the other hand, £l = Ol(cl) © 
Oi(-o), < a < n. If < a < 2, then H l £ L {l) = 0. If a = 3, H l £ L (-l) H l O L (-4) 
and ^^(l) = H l O L (-2). Then for r? G H l O L (-A) we have r? • H O L (2) ^ H l O L (-2) 
if and only if 77 = 0. Consequently, for the vanishing of H 1 ^^!) we have only to assume 
that £l 7^ or equivalently that 

^Ker(iJ 1 ^(-l)^i7 1 ^(-l)), 

which is 1- or 2-dimensional. Since by |4.3| the set of jumping lines of £ of order > 3 is 
at most 1-dimensional, £ has to avoid a subvariety of dimension < 2 or 3. If a = 4, the 
elements £ or r\ G H 1 Ol{—'o) should avoid the condition t]H°Ol(2) 7^ H 1 Ol{—?>). The 
set of these 77 is the affine cone over the rational normal curve in P/7 1 (9i(— 5). Namely, if 
s, t are homogeneous coordinates on L, we have 

i/=i 

and 

2 a l , a 2 , 82 . 13 ,2 a 3 . «4 

Then the condition t]H°Ol(2) 7^ H 1 Ol(— 3) becomes 

ai a 2 a 3 \ < 
a 2 a 3 a 4 J ~ 

So £ has to avoid another 3-dimensional subvariety in H 1 £(— 1). Finally, if a = 5, in case 
n = 5 only, the set of 77 G H x Ol{— 6) with t]H Ol{2) 7^ H 1 Ol(—4) is a cubic hypersurface 
in H 1 Ol(— 6), as can be seen by a similar argument. So £ has to avoid this hypersurface 
in H 1 £(— 1) = H 1 £l(— 1) in case a = 5. But £ has only finitely many jumping lines of 
order 5 because it is not special 'tHooft, see also |24j], p2"l . Totally £ has to avoid two 
3-dimensional and finitely many 4-dimensional subvarieties in H 1 £(— 1), in case n = 5, 
in order to satisfy the condition of the lemma. □ 




6.3. Lemma: Let £ = £ w be as in the previous lemma. Then, for a general £ G 
iJ 1 £ w (— 1), the bundle T = £& has the property that H 1 J r p(l) = for any plane P 
in P 3 . 
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Proof. By |6.2| we may assume that if ^^(l) = for any line L if £ is in a fixed open set 
of H 1 £(—l). We let £p denote the element corresponding to £ under the isomorphism 
H l £ ( — 1) = H 1 £p(— 1) for a plane P. We are going to show that H l J r p(l) = for any 
plane if £ avoids some additional subvarieties of H 1 £ (— 1). These subvarieties will be 
estimated in dimension in the following cases. 

case 1: P is a stable plane of £. In this case there is no condition on £ because we show 
that then already H 1 J r p(l) = 0. 

Proof of case 1: We prove first that h}Tp < 2. As in the previous proof we have the exact 
sequence 

i P ® H°0 P (1) ^ E X E P -> tf 1 .^ -> 0. 

Because /i°£p = 0, we have h l £p = n — 2. So, if n = 4, then /^jFp < 2. If n = 5, 
the homomorphism m(£) is zero if and only if £p is mapped to zero under the map 
H l Sp{— 1) — > Cg) H 1 £ P , where W* = H°Op(i), and the kernel of this map is the image 
of Hom(Qp(l),£p) in Ext 1 (C P (l), £ P ) iJ 1 ^ P (-l). Therefore, £ P is induced by a non- 
zero homomorphism fip(l) £p. Because both bundles are stable, tp is generically and 
then globally injective. Then C = coker(<^) has Hilbert polynomial %C(m) = m — 3 and 
is Cohen-Macaulay. It follows that C = Ol{— 4) for a line L C P. If follows that L is a 
jumping line of order 4 and that the restricted homomorphism VL p ®Ol — > 1) factors 
through (9i,(3). Then the diagram 

iPfip -^p(-I) 



If 1 ^®^) -tf^-l) 

implies that the image £l of £ in H 1 £l(— 1) is zero, and if 1 jF L (l) = 
H l £ L (l)/£ L H O L {2) = i/ 1 C L (-3) ^ contradicting Lemma This proves that 

^.T-p < 2 if £ is general. 

Let now LcPbe any line with equation z. By the assumption on £ the multiplication 
map H 1 J r p H 1 J r p(l) is surjective. Applying the bilinear map lemma of H. Hopf to 

H x {Fp{1))* <g> P°Cp(1) -> tf 1 ^)* , 

we deduce that 

h}F P {l) < h x T P - h°0 P {l) + 1 < 0. 

(when K denotes the kernel of the map, the above surjectivity condition implies that 
PiJ 1 (^(l))* xFH°0 P (l) has an empty intersection with FK in P(P 1 (J^(l))*®i7 Cp(l)), 
which implies the estimate). 

case 2: P is an unstable plane of £ but contains no jumping line of order 5. 
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In this case we have an exact sequence 

-> P -> S P -> J ZjP -> 

where Z is a O-dimensional subscheme of P of length n with iJ°X^ p(l) = 0, because P 
contains no jumping line of order n. It follows that h°Sp = 1, h°£p(l) = 3 and then 
h x £ P = n-1, ^p(I) = n-3. If /i 1 ^ < 2, then, as in case 1, h x T P {\) = 0. If /i 1 ^ > 3, 
then the multiplication map m(£), see case 1, has rank n — 4. If n = 4, then m(£) = 0, 
and we get a contradiction by the argument in case 1, which leads to h l Tp < 2. If n = 5, 
then m(£) has rank 1 and £p is annihilated by two linear forms z^, Z\ G H°Op(l). Let 
x G P be the point determined by Zq,zi and let 1{ x },p be its ideal sheaf. The standard 
resolution of this sheaf yields the exact sequence 

- H°(S P ) 2 - ff°(I {l} ,p 5p(l)) - H l £ P (-l) ^ H\8 P f. 

On the other hand, the defining sequence of Z^p and H°Op(l) = H°£p(l) implies 



h°(l {x})P ®£p(l)) = { 



2 if xeP\Z 

3 if x E Z 



Therefore, h l J-p > 3 can only occur if £p avoids at most five 1-dimensional vector sub- 
spaces of P 1 £p(— 1). Because £ has only a 1-dimensional variety of unstable planes, it 
follows that h l Tp < 2 and h l Tp{\) = for any unstable plane, if £ avoids an at most 
2-dimensional sub variety of H 1 £(— 1). 

case 3: P is an unstable plane of £ and contains a jumping line L of order n. 

In this case P^-l) ^ P^ L (-1) and /i^p(l) = n - 2 = /^(l), such that 
H l £ P (l) ^ P^l(1)- It follows that also H l F P {l) ^ H X F L {1) =0. □ 



Proof of proposition 871: Let us assume, firstly, that £ = £ w is not special 'tHooft. 
By -L5 we may assume that the multiplication map £ <S> V* —>■ H 1 £ is injective, hence 
h x T = 4, T = £ ai . By [T3]we may assume that if 1 ^-p(l) = for any plane. Therefore, the 
multiplication map H X T — > i/ 1 jF(l) is surjective for any linear form. Now an application 
of the bilinear lemma of H. Hopf to 

implies h}[T{X) < h}T — 4 + 1 = 1. If £ is a special 'tHooft bundle, it can even be shown 
that for a general £ G H 1 £{— 1) we have iP.F(l) = 0. If £ is special 'tHooft, then the 
pairing V* <8> H 1 £(— 1) — > P x £ can be defined by a 5 x 8 matrix 

/ v i t> 2 \ 

^3 ^4 «1 ^2 

f 3 W 4 t>i W 2 

f 3 W 4 «! V 2 
\ V 3 V 4 ) 
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where v\, . . . , v± is a basis of V, after choosing suitable bases of the cohomology spaces, 
see 0. Then, choosing £ = (0,0,1,0,0) in k 5 = H l £{— 1), the resulting homomorphism 
V* ® H X T{— 1) — > -ff 1 ^ is described by the matrix 







V 



\ 



V 3 Vi J 



which describes at the same time the right part 4fi 1 (l) 
of T. It follows that = in this case. 



AO of the Beilinson II monad 



7. Irreducibility of MI(5) 

In this section the irreducibility of MJ(5) is proved. The proof is mainly based on the 
properties of the sets M(4, 4)^ and the fact that for any G MJ(5) there is a £ G 
H 1 S UJ (— 1) such that (4, 4)-instanton £ a satisfies h l £cj(l) < 1. The plan of the proof is 
the following. Let us recall the notations 

M(4,4) = {u G S 2 Hl ® A 2 V* I rk(cu) = 12, u non-degenerate} 

and 

M°(4,4) = {lo G M(4,4) | H^l) = 0}. 
In addition we need a partial completion of M(4,4), by forgetting the non-degeneracy, 

M(4,4) := {u G S 2 H* A ® A 2 V* | rk(w) = 12} 
and for £ G #4, consistent with the earlier notation, 

M(4, 4) ? := {cj G M(4, 4) | Im(cu) n (f <g> V*) = 0}. 

We have 

M(4,4)f = M(4, 4) n M(4, 4) ? 
M°(4,4) c = M°(4,4) nM(4,4) € 

The space M(4, 4) is introduced for technical reasons which become apparent in Lemma 



7T| . For an element uj G M(4, 4) we only get a sheaf S u from the monad construction |2T2. 



There is the morphism 

p = res c : M(4, 4) € -> M(3, 6) = M°(3, 6) 

assigning to u the map H 3 ®V ^> H*® V*, where F 3 = Ker(f). Because rk(u>) = 12, 
this map is an isomorphism and hence non-degenerate. We are going to prove 



IRREDUCIBILITY AND SMOOTHNESS 



31 



7.1. Lemma: M(4, 4)g — > M(3, 6) is an affine bundle of dimension 54 and of fibre 
dimension 18. 

Because M(3, 6) is an open part of S 2 H^ eg) A 2 V*, it is smooth and irreducible of dimension 
36, and therefore M(4, 4)^ is smooth and irreducible of dimension 54. Then also the open 
subset M(4, 4)^ of M(4, 4)^ is smooth and irreducible of dimension 54. 

Remark: Irreducibility and smoothness of M(4, 4)g is already proved |5.7| . The bundle 
structure and the partial completion will be used to prove 

7.2. Lemma: For any £ G H\ \ {0}, dim(M(4,4) e \ M°(4,4) e ) < dimM(4,4) 5 - 2. 



Assuming the two lemmata, the irreducibility of M/(5) is achieved with the following 
arguments. The open part 

W:= U 1/(4,4), 

of M(4, 4) is smooth and irreducible by |3.4| . We let 

W° := {lo G W | H X 8 W (1) = 0} = U M°(4,4)„. 

and 

W 1 := {u G W | ^(1) < 1}. 
Then W 70 C VT 1 C W are open subsets. It follows from Lemma |7.2| , that 

codim(W 1 \ W°) < dim W 1 - 2. (1) 

Now we consider the morphisms to \— > u — res^u 

M(5,2) e -M(4,4) 
and the inverse images under r = res^, 

U\ ^ W 1 

u u 

[7° 



By Proposition |6.1| , Lemma pT| , and Proposition [4.5| , any element u G M(5, 2) is contained 
in one of the open sets U^, i.e. 

M(5,2) = UU}, 

but we don't know whether the same holds true for the sets U®. By |3 . 3 . 1| the fibres of r\ 
and r are open sets of a linear space and their dimension are 

dimrf 1 ^) = 4 + /i°^(l)<9 
dimr^ 1 ^) = 4 + h e o (l) = 8. 
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Because the difference of the fibre dimensions is at most 1, the estimate (1) implies 

dim(f/ f 1 \ U°) < dim [7° (2) 

for any £ ^ 0. It follows that 

dim(M(5, 2) \ U U?) < dim U U? = 62. 

Because any component of M(5, 2) has dimension > 25 + 40 — 3 = 62 and because U 

is irreducible, it follows that M(5, 2) is irreducible of the expected dimension 62. 

It follows now from Lemma |2.7.1| that also MI (5) is irreducible of the expected dimension 
37. We thus have 

7.3. Theorem: MI (5) is irreducible of dimension 37. 

7.4. Proof of Lemma |7.1| : We show in fact that 

M(4,4) 5 ^ M(3,6) x Horn*. (# 3 , A 2 V*). 

For the proof we need to distinguish between the linear maps A A B ® A 2 V* and the 
corresponding operators A © V A B © V* which are skew with respect to V for any two 
vector spaces A and B. Because £ is fixed, we can choose a decomposition 

H 1 ®H 3 = Ht, 

where H 3 is the kernel of f . Given w e M(3, 6) and H 3 A A 2 V* = H{ © A 2 V% we obtain 
the operator 









a 


a* 





: (ift © fig) © 7 -> (i^r © F*) © V* 



where a* denotes the dual of a with respect to H 3 and Hi. Because the upper row of 
uj is a combination of the lower row, we have rk(S5) = rk(u>) = 12. It is clear that u is 
skew with respect to V. Therefore u defines an element us € M(4, 4)f . We thus have a 
morphism 

M(3,6) x Hom(# 3 ,AV*) -> M(4,4) e 
This is even an isomorphism over M(3, 6), because if 

" = (£ S) : (#1 ® #3) - (#1* © #3*) ® AV* 
is in M(4, 4)^, we have 12 = rk(u5) = rk(a>) and it follows that 
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7.5. Proof of Lemma |7.2| : For fixed rj e HI \ {0} there is an isomorphism 

M(4,4)„ ^ M(3,6) x Hom(# 3 , AV*) =: X 



by Lemma T7I . To each pair (u,a) G X we have the simplified Beilinson II presentation 
of £(2, together with a homomorphism cr(ci!) induced by the diagram 







H 3 <g> 0(-l 



o(o)) 




0(1) 



Note here that aoe = because a is skew with respect to V, such that aouj^ 1 oj factors 
through £ a . Note further that in case (a), a) corresponds to an u e M(4, 4)^, then £ w is 
the cohomology of the monad 

0(-l)^>^A£^0(l) 

Jo 

in which case a (a) is surjective, see beginning of Section [3]. 

We let X C X be the open subset corresponding to M(4, 4) v , or defined by the surjectivity 
of a (a). In addition we let X° C X denote the open part of X where 

#V(aO(l):#%(l)^#°0(2) 

is surjective. Then 

M°(4,4) v ^ X° = X ni° 

under the above isomorphism because the cokernel of H° a (a)(1) is then isomorphic to 
H 1 S L0 (1). To prove Lemma [7.2| , it is sufficient to prove that 

codim(X\X°,X) > 2. (*) 

Now (*) will follow from the following two statements. 

Claim F: There exists an w 6 M(3, 6) such that for the fibres, X° C X Q of X° C X over 

codim(X^\X°,X tD )>2. 

Claim B: Let X C M(3, 6) be the closed subvariety of points u; for which X° = 0, i.e. 
the set of points Q for which H Q a (a)(1) is not surjective for any a. Then 

codim(S,M(3,6)) > 2. 
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7.5.1. Proof of claim B: 

a) In order to incorporate a projective curve which doesn't meet E, we enlarge M(3, 6) 
and E as follows. Let 

M(3,6) :=M(3,6)UM°(3,4). 

Then M(3, 6) C S 2 H% <g> A 2 V* consists of non-degenerate Q of rank 10 if Q G M°(3,4) 
and of rank 12 if Q G M(3, 6), and such that #"%(!) = 0. If u G M°(3, 4), the Beilinson 
II monad of E a is of the type 

0^H 3 ® O(-l) ^> #* ® ^(1) -> Q ® O -> 
where Q is the cokernel of a). 

If JF^ is the cokernel of a(a>), we have the exact sequence 

-»• £ a -»■ .F ffl -»• Q <g> O -»• 0. 



Given a second component a G Hom(if 3 , A 2 ^*), we also obtain a homomorphism ^ 
(9(1) by the diagram 



<r(a) 



H 3 ®0(-1) 



H* ® Vx^O 



^ A 



cr(o) 



which in case u G M(3, 6) coincides with £ a 



a{a) 



O(l). The surjectivity of #°cx(a)(l) 



does not depend on the choice of the factorization of a. 

We let now £ C M(3, 6) be the locus of points Co for which if°cr(a)(l) is not surjective 
for any a. By this definition we have 

E = EnM(3,6). 

b) Because Miq{H 3 ) had been shown to be an irreducible hypersurface in S 2 H^ ® A 2 V™ 
whose complement is M(3,6), see |5.4| , and because M°(3,4) is an open part of it, the 
complement of M(3, 6) in S 2 H^ ® A 2 V* has codimension > 2. In order to show that 
codim(E, M(3, 6)) > 2, we first construct an embedding k 2 s {0} — ► M(3, 6) \ E. 

c) This embedding is defined as follows. We let ex, ■ ■ ■ ,e^ be the standard basis of H^ = k 4 
and e*, . . . , el be its dual basis. We let 



Ha 



H\ ® AV* 
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be given by the matrix 



( J x 



UJ 



11 





V o 



^12 
^22 











uj u 

U 12 



o \ 



J[ 2 

"22 J 



with oj' ij: uj'lj G A 2 V* which represents the direct sum of two 2-instantons £' and £", £ w 
£' © Then if^l) = 0. For t = (t , *i) ^ we consider 

& = (-ti,0,t ,0)eif 4 * 



and its kernel 



defined by the matrix 



k 3 A> k 4 " — 



ft 



ft \ 

1 

ti 

V o o i / 



We let uj t = (/ t * (g) id) o a; o f t such that 



+ ^1^11 ^0^12 ^1^12 



UJ 



22 







t uj' l2 

tiU}" 2 u uj 22 

We may assume that both uj 22 and u;^ have rank 4 as operators V — > V*. Then by an 
elementary matrix operation we can kill £0^12 anc ^ ^1^12 i n ^ ne fi rs t row anc ^ obtain a 
matrix 







t u' 12 
tiu'{ 2 



"22 



UJ. 



22 



with i]' u 



UJ 



11 



uj^uj. 



/-1 



12^22 



12 



and rji! 



UJ 



11 



^12^22 ^12- 



Because a;' and a;" represent 2-instantons, rk(u/) = rk(u/') = 6 and therefore 

rk r)' u = rk 77^ = 2. 
But we can choose to' and a;" such that rk uj' 22 = rk u;^ = 4 and in addition 

Imr^ + Im r)ii = V*. 

Then t^n + tiVu * s an isomorphism V — > T^* for t ti 7^ 0. Therefore, with this choice of 
to' and a;", we have 

uj t e M(3, 6) for 1 7^ 
c^eM°(3,4) if t = 0orti = 0. 
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The first statement follows directly from the fact that rk oo t = 12 if t ti ^ 0. If t = or 
ti = 0, then rk uo t = 10. In that case, e.g. t\ = 0, 





^22 



and then the bundle of u t is the direct sum 



C 1 r^J 



with £' = £ w / a 2-instanton and £ w » 2 a 1-instanton (null-correlation bundle). So u>t is 
non-degenerate and H 1 £^ t (l) = 0. It follows that t 1— > uo t is a morphism 

A; 2 \ {0} -> M(3,6) 

which is an embedding by the shape of the matrix cu t . Moreover, uj t £ for any t. To see 
this, we consider the case t\ 7^ first. In that case the linear embedding k 3 — > k A can be 
considered as the kernel of a fixed £ = (1, 0, 0, 0) by the following diagram 




where & = (-fi, 0, i , 0) and £ = (1, 0, 0, 0) with 



/ \ 
1 

1 

V 1/ 



and a t 



( -1 *o/ti \ 

10 

10 

V 1 / 



If we denote 



we have 



u t := a t l, *uja t 1 



uj t = g u t g = res^u t . 

Because u t ~ u, we have S Wt = £ w and so h x £^ t {V) = 0. If cu t G M(3, 6), then cu t £. If, 
however, a) t G M°(3,4), then is the cohomology of the complex 



O(-l) 



■17* 



0(1). 



Then also in this case H^T^iV) -»■ #°0(2)Jias cokernel if^^l) = 0. Therefore w t £ £ 
for t x 7^ 0. By symmetry in t , ^ , also Q t G" £ for £ 7^ 0. Therefore we have an embedding 

k 2 \ {0} ^ M(3,6) \ £. 
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d) The map 1 1— > uj t cannot directly be used to define a line in FS 2 H£ <8> A 2 V* because u t 
is not homogeneous in t. But it can be used to construct a projective line in FH% which 
doesn't intersect a transformation of E so that we can conclude from that that S has 
codimension > 2. To do this we consider the following transformation diagram 

¥H* 4 A PHom(# 3 , H 4 )° ^FS 2 H; ® A 2 V*. 

In this diagram PHom(# 3 , H A f denotes that open set of injective maps and it is the 
principal PGL(if 3 )-bundle over the Grassmannian G(3, i^) = WH%. The morphism 9 is 
defined by 

/ (/* <g> id) o u o (f ® id) 

using the form of u in c). We let 

S C M(3, 6) C ¥S 2 H 3 * ®A 2 V* 

denote the subvarieties of which £ and M(3, 4) are the affine cones. So £ is closed in the 
open set M(3, 6) of codimension > 2. Now 

C 0- x M(3,6) C PHom(# 3 , # 4 )° 

are PGL(if3)-invariant and therefore there are an open subset U C Pif 4 and a closed 
subscheme Z C U such that 

_1 £ = 7r _1 Z and _1 M(3, 6) = vr^P. 

Moreover, there is the irreducible hypersurface M\q(H^) C FS 2 H^ ® A 2 V™ defined by 
rk 0} < 10. Its inverse image -1 Mi O (#3) is the subvariety of those (/) G PHom(P 3 , P 4 )°, 
for which (if 4 /Im(/))*®V r * — ► P 1 ^ is not an isomorphism, see |3.1|, because then a; = 8(f) 
has rank 10. It is then easily seen that this condition defines an effective divisor, which 
is O^ 1 Mio(Hz) . Because this is also PGP(P3)-invariant, there is a divisor D C PP 4 such 
that 

6- l M w (Hz) = n- l D. 
Now the family (f t ) of c) defines a linear embedding (t) 1— > ir((f t )) = (£ t ) 

Pi -> L C PP 4 * 

and by the result of c) we have 

LdU \Z and L D. 

Let K C PP 4 be the complement of P. Then L doesn't intersect Y U Z = Y U Z and so 
Z is of codimension > 2. Then also # _1 £ has codimension > 2 in # _1 M(3,6). But since 
also the complement of Q~ X M (3, 6) has codimension > 2 in PHom(P 3 , P 4 ), there is a line 
P C PHom(P 3 , P 4 ) such that 

P C 9-^(3, 6) \ _1 M and V <£ O^M^Hs). 
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We let now T = 6{L'). This is a complete curve in FS 2 H^ ® S 2 V* (not contracted to a 
point because V intersects the divisor Q~ x M\q{H^) and is not contained in it) and such 
that 

T C M(3,6) \ E. 

Then T cannot intersect the closure of £ in FS 2 H^ <g> A 2 V*. So we have shown that 
codim(E, M(3, 6)) > 2. This is equivalent to claim B. 



7.5.2. Remarks on null— correlation bundles: 

The tensors u provided for claim F are diagonal matrices of elements rj £ A 2 V* of rank 
4 (or indecomposable) which correspond to 1-instantons or null-correlation bundles (nc- 
bundles for short). Given n of rank 4, we have the defining sequence — > 0(— 1) — > 

Q 1 (1) ^ S v —>■ . Then for any a 6 AY* we obtain a homomorphism S v a ° > 0(1) as in 



7.5. 1| , using the diagram 
— 



0(-l) 



nHD 



• 



v* ®o 



y 

Oil) 



One can easily see that a (a) = if and only if a and rj are proportional in A 2 V* or 
(a) = (rj). Therefore, a non-zero image X of £ v (—l) — > O is determined by a line 
g = (a), (77) in PA 2 V* and is independent of (a) on g. We therefore write X g for the image 
of £ v (— 1) under a(a;). 

In this section we identify the Grassmannians G = G(2, V) and G(2, V*) via the Plucker 
embeddings in PA V = PA 2 V*, because A 2 V = A 2 V* ® A*V canonically. Then for a line 
/ in PV^ through (x), (y) with equations z, w we write I = (x A y) = (z A w). 



Lemma 1: Let {if) e PA 2 ^* \G and let g be a line in FA 2 V* through (rj) with associated 
ideal sheaf l g . Then 

a) // g meets the Grassmannian G in two points l\ 7^ I2, then li fl I2 — in FV and 

b) // g is tangent to G at I, then X g is the twisted double structure on the line I given 
by the tangent direction along g. 

In both cases h°2 g (2) = 4. 

Proof. Let Z be the zero scheme of X g . We have the exact sequence — > 0{ — 1) — ► 
S v -> Oil) -> O z (l) -> and by that for the Hilbert polynomial x^zM = 2m + 2. 
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This means that Z is a pair of disjoint lines or a twisted double structure on a line. In 
order to prove that X g is the ideal of the lemma, it is sufficient to determine the space 
H°I g (2) C S 2 V* by the induced diagram 

k V -* A 2 V* H%(1) 



v*®v* 



V ®v* 



S 2 V* 



where g 3 (a) ^ (rj). 

case a) Because (a), (rj), l±, l 2 are on the same line g <f_ G, we may choose a ba- 
sis eo, ... ,e3 G V with dual basis zo, . . . Z3 such that Zi = (e A ei), Z2 = (e2 A e 3 ), 
77 = z A -21 + -22 A z 3 and ct = A^ A Z\ + //z 2 A £ 3 with A ^ With this choice one 
computes that in the above diagram the image of H°a(a)(l) is spanned by the 4 forms 
-20-23; -2i-22, -2i-23 which are the generators of the ideal of l± U l 2 - 

case b) Here we may choose the same 77 but a = Z\ A z 3 such that (a) = (e A e 2 ) = / 
in PA 2 \/* = FA 2 V. With the same type of calculation we find that H°2 g (2) is spanned 
by the 4 forms zf, Z1Z3, z 2 , z^z\ + -22-23, which are the generators of the ideal of the double 
structure on I. □ 



Lemma 2: (1) Let pi,p 2 be two different points on a line I in P 3 and let l\,l 2 be two 
skew lines not meeting pi,p 2 . Then the space of quadrics containing pi,p 2 ,l 2 ,l 2 is 2- 
dimensional if and only if I doesn't meet both l\ and l 2 , and otherwise this space is 3- 
dimensional. 

(2) If Iq is a twisted double line as in Lemma 1 above, which doesn't meet I, then the space 
of quadrics containing pi,p 2 and / is 2-dimensional, too. 

Proof. By elementary computation using suitable coordinates. □ 



7.5.3. Three nc bundles: 

We consider first two nc-bundles E\ = S Vl ,S 2 = £ m with homomorphisms Si 0(1), 
and ideal sheaves T\,X 2 respectively. Then the image of H°(£i(l) © £2(1)) —> H O(2) 
under a\ + a 2 is H°Xi(2) + H°l 2 (2), and similarly for three such data (Xj,<Tj). We let 
Zi, Z 2 be the zero schemes of Ii,I 2 respectively. 

Lemma 3: Let Z 1 = Z n U l i2 and Z 2 = l 21 U l 22 with all 4 lines different. Suppose that 
either 

(i) Z\ fl Z 2 — and the 4 lines do not belong to the same ruling of a quadric, or 
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(a) Zi n z 2 = {xi,x 2 } c in an d z 2 n h 2 = 0- 

T/ien 

F%(2) nP°J 2 (2) =0. 



Proof, (i) well-known, (ii) Let Q be a smooth quadric containing in, £12 (in one ruling). 
Because Z 2i / H /12 = and l 2u fl /n = {x^}, we have hi, ^22 ^ Therefore the intersection 
of the H°X V {2) does not contain a smooth quadric. Then any Q G P°Zi(2) fl H°I 2 {2) is 
not smooth and contains the skew lines hi and Z12. So Q = Pi U P2 is a pair of planes. 
Let hi C Pi. Then Zi 2 G: P± because In n Z i2 = and so Zi 2 C P2. Then hi, I22 *t- P2 and 
hi, hi C Pi. But Z21 H Z22 = 0, a contradiction. □ 

Remark: In case (ii) the 8-dimensional space H°T 1 (2) + H°1 2 {2) equals H°Ts XljX2 \(2). 



Corollary: Let the situation be as in Lemma 3 and let £ 3 -A (9(1) be a third homo- 
morphism of an nc-bundle with ideal X 3 and zero scheme Z%. If Z 3 consists of two (then 
skew) lines Z31 , Z32 or of a twisted double line l 3 , such that the line 

hi = Xi, x 2 

doesn't meet both of hi and h 2 , then 

#°Ji(2) + #°T 2 (2) + H%{2) = H°0(2). 



Proof. By Lemma 3 the sum of the first two is H°Xr xl!X2 \(2). By Lemma 2 of [7.5.2| , 

dimH°I {xuX2} (2) n H°I 3 (2) = 2. 

□ 

7.5.4. Proof of claim F: 

a) We choose an isomorphism H 3 = k 3 and 

f m 

00= I 7] 2 

V 773 

where rji G A 2 V* are indecomposable. Let Si = £ Vi be the associated nc-bundles such 
that 

£ = £ Q = £ x © S 2 © £ 3 . 

By the definition of X a in |Z.5| this fibre is isomorphic to Hom(fc 3 , A 2 V*) and we have a 
surjective linear map 



X a -> Hom(£i © S 2 © £ 3 , 0(1)) 
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by 

(«i,a 2 ,a 3 ) (^lM, g-jM, <r 3 (a 3 )). 

The open set X® in X Q is then the inverse image of the set of those (ai, <r 2 , cr 3 ), £j 0(1), 
such that for the corresponding ideal sheaves Xj we have 

iJ J 1 (2) + #°J 2 (2) + H%{2) = H O{2). (*) 

We put Wi = Hom(^, 0(1)) and let 

Z c F(Wx ®W 2 ® W 2 ) 

be the closed subvariety of points (ox, 02, <t 3 ) for which (*) is not satisfied. Then 
codim Z > 2 implies claim F for the chosen u. 

b) In order to prove codimZ > 2 we consider the natural projection 

P(Wi © W 2 © W 3 ) \ Z' A PWi x Piy 2 x FW 3 

where Z' is the subvariety of points (<7i, o~ 2 , <x 3 ) with at least one component equal to 0. 
It is easy to see that tc is a principal bundle with fibre and group (k*) 3 /k*. We have 
Z' C Z and Z \ Z' is invariant under (k*) 3 /k*. Then K = 7r(Z \ Z') is closed and 
Z \ Z' = 7T _1 (F). Now y is the subvariety of triples ((o"i), (cr 2 ), (o" 3 )) for which (*) is 
not satisfied for the images T U {1) = Im a v . Claim F will be proved if codim Y > 2. This 
follows now from 

c) Proposition: Let (rj 2 ), (?7 3 ) G FA 2 V* \ G be in general position (not co-linear). 
Then the subvariety Y has codimension > 2. 

Proof. We let yi denote the points (rji) in PA 2 V. 

(i) Let H(y 2 ) be the polar hyperplane of y 2 with respect to the quadric G. There is point 
/ G H(y 2 ) n G such that the tangent hyperplane T\G doesn't contain y 3 . Denote by 

C{l)=GnTi{G) 

the cone of lines in PV meeting I. Then C(l) D H(y 2 ) has codimension 2. 



(ii) Choose any In G H(y 2 ) n G with y 3 G" TJ U G and such that the line g\ = y%, In meets 
G in two different points l\ 2 7^ In. We have ln,y 2 G T/ n G because in G H(y 2 ). Next we 
choose a line g 2 in T/ n G through y 2 which meets G in two different points l 2 i, l 22 . These 
lines belong to the cone C(ln) and thus each meets In in a point X\ and x 2 . We may 
assume that g 2 fl C(7i 2 ) = 0, such that / 2 i and / 22 don't meet / i2 . By this choice the 
conditions of 7.5. ^ , Lemma 3, are satisfied, and thus 

F°J 9l (2) + J f7% 2 (2) = J f7°J {:[ , iX2} (2) 

is 8-dimensional, where X 9i/ is the ideal corresponding to the line g v through y v . 

(iii) Let now P(x u ) C G be the a-plane of all lines through x u and let 

S = P( Xl ) U P(x 2 ) U (C(Z U ) n H(y 3 )). 
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Because y 3 G" 2] U G, C(ln) fl H(y 3 ) is 2-dimensional and hence S is 2-dimensional. Let 

PA 2 y\{y 3 } Ap(aVA/ 3 ) 

be the central projection. We have y 3 G" S. Then tt(S) has dimension 2 and there is 
a line L C P(A 2 V/?/3) which doesn't meet tt(S). Let 03 G L be any point and let g% 
denote the line through y 3 in PA 2 V given by 7r _1 (a3). If 03 G" Tr(H(y 3 ) fl G), the branch 
locus of n\G, then g 3 fl G consists of two different points /31, Z 32 which don't meet xi, 
Suppose that / 3 i meets In, i.e. l 3 i G C(ln) C T in G. Then Z 32 G" T) n G because otherwise 



also 1/3 G Ti n G, which had been excluded by the choice of In- By the corollary in |7.5.3 



H°X g3 (2) intersects H°T{ xltX2 y(2) in dimension 2. Then for (a J) G g v , v = 1,2,3, we have 
a point ((ai), (a 2 ), (a 3 )) G PWi x PW 2 x FW 3 \ F. If however, a 3 G 7c(H{y 3 ) n G), then 
#3 is tangent to G at some /o an d T gz is a twisted double structure on l . In this case 
lo ^ C(ln) because Iq G H(y 3 ), and so lo fl in = 0. Again by the corollary H°I ga (2) and 
H°I{ xltX2 }(2) intersect in dimension 2. 

(iv) Let now a' 3 ,a 3 G L be any two points, ^3,^3 the lines in PA 2 V through y 3 , defined 
by 7r- 1 (a£),7r -1 (a{, / ) and let (a x ) G #1 \ {yi}, (a 2 ) e 5s \ {y 2 }, (« 3 ) {^}, ("3) e 

#3 x {2/3}- Then the points 

«ai), (a 2 ), (sa 3 + to 3 '» G PWi x PW^ 2 x Plf 3 

do not belong to Y for any (s, £) 7^ (0, 0). We thus have a complete curve in the product 
space not meeting Y, i.e. codimF > 2. This completes the proof of claim F and with that 
the proof of Lemma [772] and finally the proof of theorem [7T3. □ 



8. Smoothness of MI(5) 

The induction step £ u ~> £ a will be used in order to prove that H 2 S 2 £ U] = follows from 
H 2 S 2 £ G] = even if h x £ Q {l) = 1. 

As before we use the following notation. £ will denote a 5-instanton and £ the (4, 4)- 
instanton obtained from a general £ G i/j* = H l £(— 1) by the process u; t— > u) = res^u;, 
such that £ — £ w and £ — £ a , see f 



By Proposition we may assume that there is a second element rj G H l £{— 1), linear 
independent of £, such that the multiplication map (£, 77) <g) V* — > iJ 1 ^ is injective (in fact 
an isomorphism). Let H = Ker(£) C f/5 and let f) G be the image of rj. If u = res^u;, 



then lo G M(4, 4).^, and we have H 2 (S 2 £) = 0, see [3.2| in case n = 4. Now £ appears as 
the cohomology of the monad 

-> Op(-l) -> £ -> C P (1) -> 

resulting from £. Then ,S 2 £ appears as the cohomology of the derived monad 

-> £(-1) -> S 2 £ © C -f £{1) -> 0, 
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see proof of |2.8.1| and |3.2| . Because of the vanishing conditions for instantons, we obtain 
an exact sequence 

H\S 2 £) -> H l £{\) -> H 2 (S 2 S) -> ff 2 (S 2 £) = 0. 

By Proposition |6. 1| we may assume that h l B(l) < 1. In what follows, we shall, practically, 
describe the kernel of the map if 2 (£(l))* — > H 1 (S 2 £)* and show that it is zero. Then 
it follows that H 2 S 2 £ = 0. In order to do that, we first describe the dual of if ^(l) in 
terms of its defining tensor Co, which can be done for arbitrary n. 



8.1. Lemma: Let if be an n- dimensional vector space, let uo G S 2 H* ® A 2 V* be non- 
degenerate and let £ = £ w , and interpret the elements of if ® S 2 V as linear maps if* ® 
V* — ► V which are self dual with respect to V . Then 

(a) H\£ (1))* = {7 G if ® 5V I 7 o w = 0} 

(b) f/P = PW^ zs a plane m P 3 , C V, then H\£ P (\)Y = {76^8 S 2 V | 7 o a; = 
and fm 7 C W 7 } 



Proof, a) From the selfdual monad associated to a;, one derives the exact sequence 

N ®V* ^ H* ® S 2 V* -> ff^l) -> 

and its dual 

-> if^l))* -> H ® S 2 V -f iV* <g> V. 

Interpreting the elements of N* <8> as linear maps AT — > and the elements of if ® S 2 V 
as linear maps if* ®V* ^-V = Hi®V, the homomorphism in the last sequence can be 
described by 7 1— > 7 o w, where u : N ^ H* (g) V*. 

Since A" = Im(a>), it follows that 7 o k = if and only if 7 o u — 0. 

b) If 7 G if <g> S^V and Im(7) C then 7 G if <g> S^W, i.e. 7 factorizes as 



if* <g> 1/* 



ff*® VT* 



"70 



V 



for some 70 which is selfdual with respect to W. Now restrict the monad of u to P and 
use the same argument as in a). □ 



8.2. Proposition: Any (5,2)-instanton £ on P3 satisfies H 2 S 2 £ = 0. 
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Proof, a) Let u> G M(5, 2) define £ = £ u . As before, we denote by £ the bundle £ a defined 
by uj = res^{u) for a general £ G H* . We may assume that h l £{l) < 1 by Proposition 
TL As mentioned above, we may also assume that H 2 S 2 £ = 0. 



b) Let Hi © H = H be a decomposition defined by £ with if = Ker(£). Then 

(Hx © H) © V A © iT) © V* 

decomposes as 

u; - 



e o (f o e* 


e o ^ o a* 


tt o Lp o e* 





where A" — ► ii]* © V* is a linear map and where 

F © V — ^ fP © 



N *■ N 



is the decomposition of u = u o o u*. Note that co> is symmetric, resp. skew with respect 
to if, resp. V. Similarly, any a G A 2 ff © S 2 V can be written as a linear operator 



(ff* © if*) © V* A (ffi © if) © V 



and decomposed as 






7 ' 


-7* 





with 7 G if © S 12 ^ because <r is skew with respect to if. 
c) We are now using the exact sequence 

-> H 2 {S 2 £)* -> A 2 ff © -f © if © 1/ 



of Remark |2.10| , the homomorphism being a i— ► (tow. By the above decomposition the 
condition o o u = is equivalent to the two conditions 

7011 = and 7* o e = a o u. 

In order to show that aow = implies a = 0, it is now sufficient to prove that the two 
conditions on 7 imply that 7 = 0. For then, also, ffoa = 0or(Toa) = 0, and then a — 0, 
because H 2 S 2 £ = 0, using the same sequence for £. 

d) The first condition 70^ = or 70^ = means that 

7 G tf X (£(l))* C if ®S 2 V 

by Lemma |8.1] , (a). If H 1 £(l) = 0, there is nothing to prove. So we may assume that 
7^0. By Lemma |0| we may assume that H 1 £p(l) = for any plane P = FW in 
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FV. Then, by Lemma |8.1| , (b), Im(7) <f_ W for any 3-dimensional subspace W C V. 
Therefore, 

H* <g> V* -U E x (8 1/ 
is surjective. Because 7 o w = and dim A" = 12, we obtain the exact diagram 

>- N »- H* ® V"* — ^ #1 <8> V 



AT 



#* ® V* Q 



0. 

Note that the induced projection induces an isomorphism between iV and N, because u 
and uj have the same rank 12. It follows from the decomposition of u that any vector of 
iV can be written uniquely as 

e(9) + 9 

with 9 G iV C H* <S> V*, with the isomorphism given by e(9) + 9^9, and Im(e) C £ <8> V*. 

e) We are now going to show that the condition 7* o e = a o u implies a contradiction. For 
that we choose a 2-dimensional subspace K C H* containing £, using Lemma |4.7] , such 
that A" fl (if (g) V*) contains no non-zero decomposable vector. Let £, £' be the basis of K, 
£' G 5"*. Then there are two independent linear forms z, z' G V* such that A" fl (K £g> V"*) 
is spanned by 

£ (8 2 + f ' ® 

Because £ (g> 2 under A" — > AT, we have with 9 = S, 1 ® z' that 

£ ® 2; + £' ® z' = e(£' ® -2') + £' ® ^ 
or e(£' (8) ^) = £ ® 2- Therefore, 7*(£ <8> *) = a(f <g> 2'). 

Because a G A 2 i? C^S* 2 ^ is skew with respect to H, it follows a(rj®w) G Ker(?7) ®V under 
H* ®V* H ®V iox any and any w eV*. It follows that (f ' (8 id) o 7* (f ' <g> 2) = 0, 

and dually, that 7(£' (g> 2) = 0, by considering the diagrams 



V 



H* ® V* 



£' <8>^* 



and using that 7 is symmetric with respect to V. 
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Now £' ® 2; G N and, because e has its image in £ <g> V*, there is a form z" G V^* with 
e(£' ® 2 ) = £ ® 2". Then 

(®/ + ('§zGivn (if ® V"*) 

is a second vector which is independent of £ <8> 2 + £' <g> z', a contradiction to the choice of 
K. Therefore, the condition 7* o e = a o w for 7 G ii 1 (£(1)) implies 7 = 0, which proves 
Proposition |3.2| . □ 
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